DotDFS: A Grid-based high-throughput file transfer system

DotGrid platform is a Grid infrastructure integrated with a set of open and standard protocols recently implemented on the top of Microsoft .NET in Windows and MONO .NET in UNIX/Linux. DotGrid infrastructure along with its proposed protocols provides a right and solid approach to targeting other platforms, e.g., the native C/C++ runtime. In this paper, we propose a new file transfer protocol called DotDFS as a high-throughput distributed file transfer component for DotGrid. DotDFS introduces some open binary protocols for efficient file transfers on current Grid infrastructures. DotDFS protocol also provides mechanisms for multiple file streams to gain high-throughput file transfer similar to GridFTP protocol, but by proposing and implementing a new parallel TCP connection-oriented paradigm. In our LAN tests, we have achieved better results than Globus GridFTP implementation particularly in multiple TCP streams and directory tree transfers. Our LAN experiences in memory-to-memory tests show that DotDFS accesses to the 94% bottleneck bandwidth while GridFTP is accessing 91%. In LAN disk-to-disk tests, comparing DotDFS protocol with GridFTP protocol unveils a set of interesting and technical problems in GridFTP for both the nature of the protocol and its implementation by Globus. In the WAN experimental studies, we propose a new idea for analytical modeling of file transfer protocols like DotDFS inspired by sampling, experimentation and mathematical interpolation approaches. The cross-platform and open standard-based features of DotDFS provide a substantial framework for unifying data access and resource sharing in real heterogeneous Grid environments.